Selling Real-World Health Care Research to Reluctant Buyers-Evidence-Based Education or Marketing a Defective Product?

In December 2004, the editors of BMJ announced a new policy requiring submission of a priori research protocols with manuscripts reporting the results of randomized controlled trials (RCTs). Under the new policy, RCTs would have to be registered at their outset with “a suitable trial registry,” and manuscripts lacking a registered protocol would no longer be sent to peer review. The editors explained that this decision had been made because their “experience of chasing authors for trial protocols, when we have suspected deviation in the protocol or found it hard to fathom what the authors set out to do,” had been “miserable.”1 As occasionally miserable editors, we too are familiar with the experience of seeking information from an author who is unable or unwilling to provide it; and evidence is mounting that we are not alone. As we and others have observed previously, the practices of selective reporting of study findings, publication planning, and other forms of misconduct are, sadly, reportedly endemic in health care research.2,3 Studies that use observational or “real-world” data, particularly pharmacoeconomic modeling and retrospective analyses of administrative databases, are particularly vulnerable to manipulation; it is especially easy to make post hoc changes to a planned protocol behind closed doors when only claims data and hypothetical patient populations, not prospectively studied human subjects, are involved.4 Thus, to the extant problem presented by Brixner et al. in their commentary on use of real-world data in this issue of JMCP5— that decision makers are sometimes reluctant to rely on analyses of real-world data—one reasonable response is that the most reluctant “buyers” of research may well be the best informed. After all, public denunciations of the “scandal of poor epidemiological research” by von Elm and Egger in 2004,6 and the “scandal of poor medical research” by Altman in 1994 and again in 20027,8 are well-known to anyone who has been following health care research even peripherally, and have spawned dozens of publications on how to improve a demonstrably inadequate pool of knowledge about the economic and clinical outcomes of health care interventions.2 Yet, we also know that many—perhaps even most—researchers are “playing by the rules,” endeavoring to provide accurate information, and producing high-quality work. So for a typical decision maker, the question becomes how to distinguish between accurate and inaccurate information. This is essentially the question raised by Brixner et al., and it is an important one. The proposals offered by the March 2008 participants in a roundtable discussion of real-world data, whose views are reported in the Brixner et al. commentary, merit consideration. Nonetheless, currently these ideas appear to generate more new questions than specific guidance. More troubling is the possibility that the effort to promote use of real-world data by decision makers may detract from ongoing efforts to improve the quality of information provided to them.

I n December 2004, the editors of BMJ announced a new policy requiring submission of a priori research protocols with manuscripts reporting the results of randomized controlled trials (RCTs). Under the new policy, RCTs would have to be registered at their outset with "a suitable trial registry," and manuscripts lacking a registered protocol would no longer be sent to peer review. The editors explained that this decision had been made because their "experience of chasing authors for trial protocols, when we have suspected deviation in the protocol or found it hard to fathom what the authors set out to do," had been "miserable." 1 As occasionally miserable editors, we too are familiar with the experience of seeking information from an author who is unable or unwilling to provide it; and evidence is mounting that we are not alone. As we and others have observed previously, the practices of selective reporting of study findings, publication planning, and other forms of misconduct are, sadly, reportedly endemic in health care research. 2,3 Studies that use observational or "real-world" data, particularly pharmacoeconomic modeling and retrospective analyses of administrative databases, are particularly vulnerable to manipulation; it is especially easy to make post hoc changes to a planned protocol behind closed doors when only claims data and hypothetical patient populations, not prospectively studied human subjects, are involved. 4 Thus, to the extant problem presented by Brixner et al. in their commentary on use of real-world data in this issue of JMCP 5that decision makers are sometimes reluctant to rely on analyses of real-world data-one reasonable response is that the most reluctant "buyers" of research may well be the best informed. After all, public denunciations of the "scandal of poor epidemiological research" by von Elm and Egger in 2004, 6 and the "scandal of poor medical research" by Altman in 1994 and again in 2002 7,8 are well-known to anyone who has been following health care research even peripherally, and have spawned dozens of publications on how to improve a demonstrably inadequate pool of knowledge about the economic and clinical outcomes of health care interventions. 2 Yet, we also know that many-perhaps even most-researchers are "playing by the rules," endeavoring to provide accurate information, and producing high-quality work. So for a typical decision maker, the question becomes how to distinguish between accurate and inaccurate information.
This is essentially the question raised by Brixner et al., and it is an important one. The proposals offered by the March 2008 participants in a roundtable discussion of real-world data, whose views are reported in the Brixner et al. commentary, merit consideration. Nonetheless, currently these ideas appear to generate more new questions than specific guidance. More troubling is the possibility that the effort to promote use of real-world data by decision makers may detract from ongoing efforts to improve the quality of information provided to them. Research] Task Force on the Use of Real-World Data," was the formation of an "independent body or review process." The body would "be formed as a consortium of experts giving access to a broad range of resources and expertise for an audit, review, or quality certification process." 5 We wonder how such a group would differ from the available pool of journal peer reviewers who, as experts in particular topic areas, are already tasked with screening the quality of research articles. Certainly, any consortium of experts would be faced with, and challenged by, the same lack of transparency in research reporting that has by now become infamous among journal peer reviewers, editors, and methodologists. A key task of peer review-to ensure that limitations of published work are transparently disclosed in terms that are relevant to journal readers-depends in part on the cooperation of authors, and most journal editors would acknowledge that some authors are more cooperative than others.
Brixner et al. mention the efforts of the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network, formed in May 2006 to "improve the quality of scientific publications by promoting transparent and accurate reporting of health research." 9 They opine that the EQUATOR network's endeavors would be synergistic with the efforts proposed by the roundtable. However, at this early stage of development, Brixner et al. offer little specific information about how the process that they propose would incorporate the EQUATOR guidelines, or how an expert consortium might be more successful in encouraging transparency than are organizations and journals that are already measuring research reports against the EQUATOR standards. Brixner et al. do raise the possibility of a voluntary registry of observational studies, similar to the registries now required by most journals for the submission of manuscripts that describe clinical trials. This idea has merit, but either voluntary or mandatory registration would pose unique challenges; voluntary registration would lack "teeth," while journals that impose mandatory registration of observational studies would at least temporarily experience a reduction in the flow of manuscript submissions.

Public Information Campaign-How Great Is The Need?
A second key proposal advanced by the roundtable participants was a "process to achieve dissemination and acceptance of an assessment tool," essentially a public information campaign that would "increase public awareness of the need for quality assessment of observational evidence and subsequently, the acceptance of observational studies meeting defined quality standards to be used in the decision-making process." 5 The window of opportunity for public dissemination of this information appears to be open-but only by the slightest crack. Deficiencies in the research literature have been publicly acknowledged for about 15 years, and EQUATOR standards have been adopted by a wide variety of journals, including Annals of Internal Medicine, BMJ, JAMA, PLoS Medicine, the Journal of Clinical Epidemiology, and others. 10 EQUATOR standards have been endorsed by the Council of Science Editors, and guideline documents promulgated by EQUATOR are referenced by the International Committee of Medical Journal Editors in their uniform guidelines for manuscript submission. 11,12 Certainly these efforts must be known among those who routinely read health care journals.
However, it is possible that many decision makers will not fall into that category. The roundtable's work, which is in its early stages, has to date included only informal assessments of informational needs that were expressed by small numbers of meeting participants and by a convenience sample (N = 70, including an unknown number of decision makers) responding to a draft ISPOR report during an open comment period. Before launching a widespread informational campaign, Brixner et al. should be encouraged to conduct a more quantitative and systematic evaluation of the informational needs of decision makers. It might even be wise to "quiz" decision makers to assess gaps in their knowledge of observational research methods and pharmacoeconomic modeling. For example, in advance of the first EQUATOR meeting, network organizers "searched literature to identify published reporting guidelines and surveyed authors to examine how the guidelines were developed and to identify problems encountered during the development" and used "the survey results and meeting discussions [to help us prioritize] main activities that were necessary for a successful start of the EQUATOR Network's efforts to improve the quality of reporting of health research." 13 This kind of systematic, evidence-based approach is most likely to result in a process that will address the needs of decision makers.
An educational effort should also reflect a realistic acknowledgement of the time and resources available to a typical health care decision-maker. Although the key elements of high-quality research-such as transparency, minimization of bias, and a presentation that is sufficiently detailed to facilitate replication of study methods-are promulgated universally by guideline documents, 2 putting these ideas into practice sometimes requires time and expertise. The expertise required for use of research guideline documents-at about the level of upperclass undergraduate or beginning graduate school research methods and statistics classes-should not exceed the ability of any researchers seeking journal publication, but could potentially be too far outside the time or expertise available to a decision maker, even after a brief educational intervention. For example, a common error seen in research reporting is the use of an underpowered sample size to compare 2 treatment approaches, followed by the assertion of a researcher that the treatments are equivalent. An author or journal peer reviewer can easily obtain the necessary power calculation tables and determine if the sample size was adequate for the task, but asking for this level of effort from a health plan executive/decision maker is probably unrealistic. However, it is realistic to train decision makers to recognize an even more common error, the description of a clinically meaningless result as important based solely on statistical significance. For example, a difference of 1 percentage point in medication possession ratio, representing only about 4 days of additional pharmacotherapy per year, is clearly unimportant whether it is statistically significant or not.

Standardized Quality Instrument-A Worthwhile Endeavor?
An additional key element in the strategy proposed by the roundtable participants in the report by Brixner et al. is a "standardized instrument for quality assessment," which would represent "criteria that are recognized as key indicators" of quality. For the instrument to be "user-friendly," Brixner et al. posit, it would "identify the 10 most important factors from the decision maker's perspective." However, they also indicate that the instrument would take "the form of a modular assessment tool with different axes by (a) study objective: economic impact or cost-effectiveness, health outcomes, patient reported outcomes; and (b) study type: model, clinical prospective study, or retrospective data analysis." This proposal is the weakest and least evidence-based of those advanced by Brixner et al.
First, it appears that Brixner et al. may be proposing too much for just 1 document. When the Agency for Healthcare Research and Quality (AHRQ) undertook a systematic review of checklists used to rate the strength of scientific evidence in 2002, it wisely opted not to attempt a single checklist for all study types. "In the worst case," the AHRQ observed, "combining all such systems into a single evaluation framework risked nontrivial confusion and misleading conclusions, and [we] were not willing to take the chance that users of this report would conclude that 'a single system' would suit all purposes. That is clearly not the case." 14 Second, the most important potential need, education, does not require a new checklist. In an era of scarce resources, Brixner et al. should be mindful of the degree to which their work duplicates that already being undertaken, or already completed, by EQUATOR. For example, among the EQUATOR network's primary objectives are the development of "a comprehensive webbased 'Resource Centre' providing up-to-date information, tools and other materials relating to reporting health research" and "training courses for editors, peer reviewers and researchers, and other educational activities raising awareness and importance of reporting guidelines." 15 A library of standards for numerous types of study reports-including RCTs, observational research, analyses of health care interventions, meta-analyses, and moreis already available on-line, 16 and training courses have begun. 9 Since sensible and high-quality guideline documents already exist for nearly every conceivable major type of study, 2 expending effort to create yet another tool seems unproductive at best, and at worst has the clear potential to become a pyrrhic attempt at an ineffective and possibly misleading "one size fits all" approach.

Our Continued Plea for Transparency
We continue to believe that the best hope for improving the quality of research evidence available to decision makers lies in a strengthened journal peer review system. Although far from perfect, the system of editorial and peer review has already been bolstered by the efforts of the EQUATOR network and others to add transparency to research reporting and by the increased "disinfectant/sunshine" that has resulted from public attention to incidents of research misconduct. Creating a new review body or expert consortium, composed of the same people who make up the current peer reviewer pool, is unlikely to be productive and at worst may divert attention from current quality improvement efforts.
The proposal from the roundtable session reported by Brixner et al. that has the most potential is the education of decision makers in the critical review of studies employing real-world data. However, such education should (a) be based on a systematic and quantitative assessment of the informational needs of decision makers; (b) openly acknowledge the scope of the problem of poor-quality research; (c) train decision makers in review methods that can be used to refute rather than accept poor quality; and (d) acknowledge the limitations of time and resources available to a typical decision maker.
Finally, the process proposed by Brixner et al. should be designed so as to minimize duplication of effort. In that regard, the clearest area for improvement in Brixner's et al.'s proposals involves the standardized checklist, which is almost entirely duplicative of work already completed by others.
Brixner et al. address an issue that has become increasingly important in managed care-how to encourage decision making that is based on high-quality evidence. However, the process currently advocated by the roundtable meeting participants poses the risk of inadvertently encouraging decision makers to accept poor-quality work-providing a false sense of security about published real-world research evidence instead of facilitating a critical assessment of its strengths and weaknesses. To reduce this risk, measurement of the unmet informational needs of managed care decision makers is essential to establish a base of evidence before "promoting the utilization of real-world data." 5